

^ 10/526552 

Rec'd TO/FFQ 03 MAR 20C1 5 



PRIORITY 
I DOCUMENT 

! SUBMITTED OR TRANSMITTED ^ 
COMPLIANCE WITH RULE 17.1(a) OR (b) 




Patent Office 
Canberra 



I, JONNE YABSLEY, TEAM LEADER EXAMINATION SUPPORT AND 
SALES hereby certify that annexed is a true copy of the Provisional specification 
in connection with Application No. 2002951244 for a patent by TELSTRA NEW 
WAVE PTY LTD as filed on 06 September 2002. 




# 



Regulation 3.2 



TELSTRA NEW WAVE PTY LTD 



AUSTRALIA 
Patents Act 1990 

PROVISIONAL SPECIFICATION 

for the invention entitled: 

"A DEVELOPMENT SYSTEM FOR A DIALOG SYSTEM" 



The invention is described in the following statement: 



PAOPETODB Www dialog system piov.doc< September. 20(0 

-1- 

A DEVELOPMENT SYSTEM FOR A DIALOG SYSTEM 
FIELD OF THE INVENTION 

The present invention relates to a development system for a dialog system, and in 
5 particular to a system and process for developing a natural language interactive dialog 
system. 

BACKGROUND 

A dialog system has a text or audio interface, allowing a human to interact with the system. 

10 Particularly advantageous are 'natural language' dialog systems that interact using a 
language syntax that is 'natural' to a human. A dialog system is a computer or an 
Interactive Voice Response (IVR) system that operates under the control of a dialog 
application that defines the language syntax, and in particular the prompts and grammars 
of the syntax. For example, IVRs, such as Nortel's Periphonics™ IVR, are used in 

15 communications networks to receive voice calls from parties. An IVR is able to generate 
and send voice prompts to a party and receive and interpret the party's voice responses 
made in reply. However, the development of a dialog system is cumbersome and typically 
requires expertise in both programming and the development of grammars that provide 
language models. Consequently, the development process is often slower than desired. 

20 

One approach to reducing the time and expertise of developing natural language dialog 
systems is to use processes whereby a relatively small amount of data describing the task 
to be performed is provided to a development system. The development system can then 
transform this data into system code and configuration data that can be deployed on a 
25 dialog system, as described in International Patent Publication number WO 00/78022, A 
Method of Developing An Interactive System ("Starkie"). However, one difficulty of this 
process is that the development system needs to make numerous assumptions, some of 
which may result in the creation of prompts that, while understandable to most humans, 
could be expressed in a manner more easily understood by humans. For example, a prompt 




WOPETODB VAtow (Balej lyxtem pttnuloc-6 Scptanbcr. 2002 



may be created that prompts a person to provide the name of company whose stocks they 
wish to purchase. The development system might create a prompt such as "Please say the 
company" whereas the phrase "Please say the name of the company whose stocks you 
wish to purchase" may be more understandable to a human interacting with the dialog 
5 system. 

Another approach, described in Starkie, for reducing the time and expertise requirements 
for developing a natural language dialog system is to use processes whereby developers 
provide examples of sentences that a human would use when interacting with the dialog 
10 system. A development system can convert these example sentences into a grammar that 
can be deployed on a computer or IVR. This technique is known as grammatical 
inference. Successful grammatical inference results in the creation of grammars that: 

(i) cover a large proportion of the phrases that people will use when interacting 
with the dialog system; 
1 5 (ii) attach the correct meaning to those phrases 

(iii) only cover a small number of phrases that people won't use when 
interacting with the dialog system; and 

(iv) require the developer to provide a minimal number of example phrases. 

20 The use of grammatical inference to build a dialog system is an example of development 
by example, whereby a developer can specify a limited set of examples of how the dialog 
system should behave, rather than developing a system that defines the complete set of 
possible examples. 

25 Thus a development system can be provided with a list of interactions between a human 
and a dialog system using a notation that lists the sentences in the order they are spoken or 
written, indicating whether it is either the dialog system or the human that is speaking (or 
writing). This is referred to as an example interaction. Similarly, an example interaction 
can be defined by recording or transcribing the interactions between two or more humans. 

30 A benefit of this technique is that example interactions are understandable to anybody who 
understands the language contained within them. In addition most people would be capable 
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of creating example interactions of desired behaviour. There is also the benefit that 
example interactions describe specific behaviours, given a set of inputs, and therefore 
provide test cases for the behaviour of the dialog system. As they document specific 
behaviour, there is also a reduced risk of errors being introduced in the specification of the 
5 dialog system for the given behaviour listed in the example interactions. Example 
interactions are also ideal forms of documentation to describe the behaviour of the dialog 
system to others. 

Example interactions can be annotated to include high level descriptions of the meaning of 
10 a sentence. This annotation might include, the class of the sentence, and any key pieces of 
information contained in the phrase known as slots. For example the sentence "I want to 
buy three hundred acme bolt shares" might be annotated to signify that the class of the 
sentence is buy_stocks as opposed to sell_stocks, and that the quantity slot of the sentence 
is 300, while the stockname slot is "acme bolt limited." 

15 

An example interaction can be converted into a model that describes a sequence of 
sentence classes. One example of such a model is a state machine. A state machine is a 
model that causes an action or output to take place, depending upon the input it receives. A 
state machine can react differently to the same input, depending upon its current state. The 
20 state of a state machine can also change depending upon the input it receives. For example, 
a dialog system may respond differently to the sentence "yes i'm sure" depending upon 
whether it has just asked the question "are you sure you want to quit" or the sentence "are 
you sure that you want to buy three hundred shares of acme bolts". 

25 Grammatical inference techniques are used for creating a state machine that can generate a 
sequence of symbols, using an example set of sequences. These techniques are typically 
applied to the creation of a class of grammars known as regular grammars. As a result a 
state machine that defines all of the valid sequences of sentence classes in a dialog system 
can be inferred, from a limited set of example interactions. To do this with a minimal 

30 number of training examples requires some assumptions to be made. In particular, this 
approach suffers from the following difficulties: 




(i) The example dialogs need to be constrained in some way to allow easy 
generalisation. For instance, it is extremely difficult, if not impossible, for a 
current state of the art development system to automatically build a dialog 
system from a set of examples interactions between two humans. 
5 (ii) Specifying a dialog system using only a set of example interactions may 

require a large number of example interactions. The difficulty in doing this 
may outweigh any benefit of reduced speed of development. 

(iii) Sentences in the set of example interactions need to be annotated 
consistently. 

10 (iv) A set of example interactions contains only information that is visible to 

someone interacting with the dialog system. Example interactions contain 
sentences created by the dialog system and the human interacting with it, 
but not the sequence of internal interactions required to make decisions as 
to what sentences should be spoken by the dialog system. 

15 

Thus a development system for dialog systems that uses only example interactions is 
unlikely to meet the objectives of reduced development time and expertise. It is desired to 
provide a system and process for developing a dialog system that alleviate one or more of 
the above difficulties, or at least provide a useful alternative to existing development 
20 systems and processes. 

SUMMARY OF THE INVENTION 

In accordance with the present invention, there is provided a process for developing a 
25 dialog system, including automatically generating a plurality of sample interactions 
representative of interactions between said dialog system and a user of said dialog system 
on the basis of definition data for said dialog system. 



Preferably, said sample interactions include interaction text and application data. 

30 
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Preferably, the process includes modifying said sample interactions on the basis of 
interaction data received from a developer. 

Preferably, said definition data includes prompt data, and the process includes modifying 
5 said prompt data on the basis of said interaction data. Preferably said prompt data is in a 
grammar format. 

Preferably, the process includes generating said definition data on the basis of specification 
data received from a developer. 

10 

Preferably, said definition data is generated on the basis of said specification data, and a 
predetermined dialog application or at least one dialog application template. 

The present invention also provides a development system having components for 
1 5 executing the steps of any one of the above processes. 

The present invention also provides program code for executing the steps of any one of the 
above processes. 

20 The present invention also provides a computer readable storage medium having stored 
thereon program code for executing the steps of any one of the above processes. 

The present invention also provides a development system, including a scenario generator 
for generating a plurality of sample interactions representative of interactions between a 
25 dialog system and a user of said dialog system on the basis of definition data for said 
dialog system. 

Preferably, the development system includes a scenario editor for editing said sample 
interactions. 

30 
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Preferably, the scenario generator generates graphical user interface components for 
modifying sample interactions representative of interactions between said dialog system 
and a user of said dialog system. 

5 Preferably, the development system includes a dialog learner for updating said prompt data 
on the basis of the modified sample interactions. 

Preferably, the development system includes an application builder for generating said 
definition data on the basis of specification data provided by a developer. 

10 

The present invention also provides a graphical user interface for use in developing a 
dialog system, said interface including graphical user interface components for modifying 
sample interactions representative of interactions between said dialog system and a user of 
said dialog system. 

15 

Preferably, said graphical user interface components include a component for deleting a 
selected sample interaction. 

Preferably, said graphical user interface components include a component for a developer 
20 to select a predetermined reason why said selected sample interaction was deleted. 

Preferably, said graphical user interface components include a component for defining a 
question and an answer to said question for display to a user of said dialog system. 

25 Preferably, said graphical user interface components include a question text component for 
defining question text and an answer text component for defining answer text. 

Preferably, said graphical user interface components include a component for selecting 
from a list of predetermined answers to said question. 



30 
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Preferably, said graphical user interface components include a component for specifying 
that said question can be asked at any time. 

Preferably, said graphical user interface components include a component for specifying 
5 that said question can be asked during a selected sample interaction. 

The present invention also provides a dialog application generated by the development 
system. 

1 0 BRIEF DESCRIPTION OF THE DRAWINGS 

Preferred embodiments of the present invention are hereinafter described, by way of 
example only, with reference to the accompanying drawings, wherein: 

Figure 1 is a schematic diagram of a preferred embodiment of a natural language 
application development system connected to an IVR via a communications network, with 
15 the IVR connected to a telephone via a telecommunications network; 

Figure 2 is a block diagram of the natural language application development 

system; 

Figure 3 is a flow diagram of an application development process of the natural 
language application development system; 
20 Figure 4 is a flow diagram of an simulate and refine application process of the 

application development process; 

Figure 5 is a flow diagram of a scenario generation process of the simulate and 
refine application process; and 

Figures 6 to 8 are screenshots of windows generated by the natural language 
25 application development system. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

As shown in Figure 1, a natural language application development system 100 can be 
connected to a VoiceXML-enabled interactive voice response system (IVR) 102 via a 
30 communications network 104. The development system 100 executes a natural language 
application development process which allows an application developer to develop a 
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natural language dialog system using a graphical user interface of the development system 
100. The development system 100 generates a dialog application that can be installed on 
the IVR 102 via the network 104 to create and configure the dialog system. A standard 
telephone 106 can be used to access the IVR 102 via the public switched telephone 

5 network (PSTN) 108, allowing a user of the telephone 106 to interact with the natural 
language dialog system by speaking into the telephone 106 to provide speech input to the 
dialog system in response to voice prompts provided by the dialog system. Alternatively, 
the natural language application development system 100 can generate a natural language 
dialog application for execution by a standard computer system to provide a dialog system 

1 0 that can accept speech (z. e. , audio) input or text input. 

In the described embodiment, the natural language application development system 100 is 
a standard computer system, such as an Intel™-based personal computer executing a 
Microsoft Windows™ operating system, and the natural language application development 

15 process is implemented by software modules, shown in Figure 2, stored on non-volatile 
memory of the development system 100. However, it will be apparent to those skilled in 
the art that at least parts of the natural language application development process or 
modules can be alternatively implemented by dedicated hardware components such as 
application-specific integrated circuits (ASICs). The IVR 102 may be a Nortel 

20 Periphonics™ IVR. The IVR 102 executes a dialog application that includes VoiceXML 
language elements. However, the dialog application could include any language that can be 
used to define a spoken dialog application, such as VOXML or a proprietary language. The 
network 104 may be any communications network that enables the dialog application to be 
loaded onto the IVR 102, such as an Ethernet LAN or WAN. 

25 

As shown in Figure 2, the application development system 100 includes an application 
wizard 202, an application builder 204, a scenario generator 206 with a simulator 208, a 
dialog learner 212, a grammar learner 214, a prompt learner 218, and a code generator 216. 
The development system 100 also includes an application library 220 and application 
30 templates 222. The development system 100 constitutes an integrated development 
environment (IDE) for the development of natural language applications. 
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The development system 100 executes an application development process, as shown in 
Figure 3, that begins at step 302 when the developer defines application specification data 
describing the dialog application at a high level using the application wizard 202. This 
includes defining operations or tasks that are to be performed by the dialog system being 

5 developed, by providing the name of each corresponding task, along with information that 
needs to be collected and the information that is created as a result of executing the 
operation, along with the type of each item of information. For example, the developer can 
specify that the dialog system is a stock or share system with a "buy" and a "quote" 
operation. The "buy" operation requires a stock name, a stock quantity and a stock price. 

10 The quantity is of predefined type integer and the price is of predefined type money. The 
values that the stock name can take are defined by providing a list of all available stock 
names. The developer can also specify a number of predefined options for the dialog 
system, for example, the developer can specify that the dialog system is not protected by a 
personal identification number (PIN) and does not allow callers to leave messages or to 

1 5 . transfer to an operator. 

At step 308, the application builder 204 generates an application definition 224, as 
described below, from the high level specification data on the basis of rules defined by 
application templates 220, as described in Starkie. Alternatively, the application definition 

20 224 can be based on an existing dialog application by the developer selecting from a list of 
predefined applications stored in the application library 222. For example, the application 
library 222 may define a telephone ordering system wherein a user can list products that 
can be purchased along with their prices and available quantities. The application builder 
204 can generate the application definition 224 by adding new code generated from the 

25 specification data and an application template 220 to a copy of the selected application 
from the application library 222. If the selected telephone ordering system includes much 
of the required functionality of the desired stock application, this can reduce the 
development time considerably. 

30 As shown in Figure 2, the application definition 224 includes a description of a dialog state 
machine 226, recognition grammars 228, and prompt models 230. The dialog state 
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machine 226 is represented in voice extensible markup language, also known as 
VoiceXML or VXML, as described at http://www.voicexnil.org, whereas the recognition 
grammars 228 and prompt models 230 are represented in a grammar notation, as described 
in Starkie. Hence the prompt models 230 are also referred to as prompt grammars 230. As 
5 described below, the prompt grammars are subsequently converted into part of the 
application 236 that converts variables, and a prompt name, into either a text string to be 
sent to a text to speech system of the IVR 102, or a list of pre-recorded audio files that are 
concatenated together. The dialog state machine 226 defined in VXML can be extremely 
complex due to the inclusion of a general purpose programming language within the 
10 VXML standard. 

Once the initial application definition 224 has been generated, it can be simulated and 
refined at step 306 until the developer is satisfied that it meets the desired requirements. As 
shown in Figure 4, the simulate and refine application process begins by generating a set of 

15 covering prompts 232 from the prompt grammar 230 at step 402. The covering prompts 
232 are such that for every rule in the prompt grammar 230 there is at least one 
corresponding example sentence, as described below. Alternatively, some rules in the 
prompt grammar extracted from the application library 230 can be annotated prior to their 
addition to the application library 222 to indicate that they are not to be modified by 

20 prompt learning, as described below, in which case no covering examples are required for 
these rules. 

In addition to generating covering prompts, the application builder 204 also generates a set 
of covering examples of the recognition grammar to be used for grammar learning, as 
25 described in Starkie. 

The application wizard 202 then invokes the scenario generator 206 to generate scenarios 
or example interactions 234 at step 406. The scenario generator 206 includes a simulator 
208 which can interpret complex VXML scripts. The simulator 208 can operate in two 
30 modes. In a first mode, the simulator 208 interacts with the developer using text input and 
stores the interaction in the form of an example interaction 234. In a second mode, the 
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simulator 208 is connected to the scenario generator 206, and the scenario generator 206, 
rather than the developer, provides the input to the simulator 208. The scenario generator 
206 is closely coupled to the simulator 208 in order to determine which grammars are 
active at any point in a dialog. The scenario generator 206 can also determine the scope of 
5 a grammar, which indicates whether the grammar is active all the time, or only when it is 
in a particular dialog state. The scenario generator 206 then attempts to provide enough 
input to the simulator 208 to create a set of example interactions 234 that are broad enough 
to give a good representation of the behaviour of the dialog system. Although the natural 
language application development system 100 is used to develop dialog systems that are 

10 mixed initiative applications (i.e., applications capable of interpreting an input phrase 
containing multiple pieces of information, and/or accepting input phrases in any sequence, 
as provided by the speaker), example interactions that are computer directed (i.e. y 
containing a single piece of information and only being provided by the speaker when 
requested by the application) provide a greater coverage of the number of prompts than 

15 example interactions that are mixed initiative. Consequently, the scenario generator 206 
first attempts to create example interactions that contain only computer directed dialogs. 
When executed a second time, the. scenario generator 206 attempts to create mixed 
initiative examples. The scenario generator 206 generates a similar number of mixed 
initiative examples as the number of computer directed examples. 

20 

The aim of the scenario generator 206 is to generate the smallest number of example 
interactions 234 that cover the majority of the application. The scenario generator 206 
takes as its input complex VXML 226 and grammars 228, 230. In addition, the VXML 226 
can make reference to logic stored elsewhere. For example, the VXML 226 can call 
25 common gateway interface (CGI) scripts, and it can access a database. Although the 
VXML 226 can be extremely complex, simplifying assumptions can be made that result in 
a high probability of creating a good covering set of example interactions, even if the 
assumptions are not true. 

30 Firstly it is assumed that the VXML 226 can be represented by a Markov model. A 
Markov model is a model that represents a sequence of events. After one particular event 
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has occurred, only a finite number of other events can occur after it, with a given 
probability. Markov Models are useful for modelling systems in which the sequence of 
events is random, or when the sequence of events follows some pattern, but where the 
pattern is too difficult to model Although Markov Models contain probabilities, the 
5 probabilities of the model are ignored by the scenario generator 206, In the model used by 
the scenario generator 206, an event is a line in an example transaction 234. Each line in 
the example interaction 234 represents an event, either the dialog system performing a 
task, or the speaker performing a task. An example interaction is given in appendix B using 
the notation described in appendix A. Each line in the example interaction can be described 
10 by its event type. 

For speech recognition events, the name of the event is the same as the name of the 
corresponding grammar fragment, which is comprised of the name of the grammar being 
activated and the top level nonterminal within that grammar, for example: 

15 xx @airline: : . Ask_timetablesf orm_destination_city", 

where "airline" is the name of the grammar and 
Ask_ timetablesform_destination_city" is the name of the top level 
nonterminal. 

20 For prompt events, the name of the event is the name of the prompt. Where a prompt is 
also described by a prompt grammar, the name of the prompt event is the name of the 
prompt grammar fragment, for example: 

"Qairline . prompts : : . Ask_timetablesf orm_city_of_departure" 

25 A number of names are reserved. For example, the name " @ speaker : : " is given to an 
event that represents the playing of prompts that are not represented by a prompt grammar. 
The names "Qstartcall : : ", "@endcall : : and "©hangup: : * are given to 
the events where the dialog system answers a call, the dialog system ends a call, and when 
the speaker hangs up, respectively. Names of the form "@record: :NAME-OF- 

30 VARIABLE " are given to events where a developer's response is recorded. 
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Rather than storing a Markov model to describe the dialog, the scenario generator 206 
includes the following data structures: 

(i) a list of all valid speech recognition events (Grammar Fragments), including a 
5 flag indicating whether the response to that event is expected to be the same 

regardless of the prompt played before it, and a count of the number of times 
that response has been given to the dialog system (i.e., the simulator 208); 

(ii) a list of all valid speech recognition events (Grammar Fragments) that can be 
spoken after each prompt is played; and 

1 0 (iii) a list of example interactions. 

The third structure provides the example interactions 234 that are subsequently output by 
the scenario generator 206, and is used to determine what responses the scenario 
generator 206 should give at particular points of the dialog. 

15 When generating menu driven example interactions, the scenario generator 206 operates in 
one of two submodes: a random walk mode, and a playback mode. The scenario generator 
206 initially starts in random walk mode, with all three data structures empty. The VXML 
state machine dialog 226 is executed by the simulator 208 and the output generated is 
stored in the list of example interactions. 

20 

The example interactions 234 are generated by a scenario generation process, as shown in 
Figure 5. At step 502, the simulator 208 provides a question of the dialog to the scenario 
generator 206, together with a list of allowable responses or grammar fragments, along 
with the scope (local or global) of those grammar fragments. The scenario generator 206 
25 uses this information to update its lists of valid speech recognition events. A grammar 
fragment is classified as global if it is always active and can be spoken at any point. The 
scenario generator initially assumes that the dialog system responds to global events in the 
same way, regardless of the question being asked, whereas the dialog system is expected to 
respond to local events in a different way depending upon the question being asked. 



30 
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At step 506, the scenario generator 206 randomly selects a grammar fragment that has not 
been considered in that state, i.e. 9 a fragment that is not represented in the first list with a 
non-zero count value. At step 508, a phrase is randomly generated using the selected 
grammar fragment and is passed to the simulator 208 and at step 510, the list of used 
5 grammar fragments is updated to reflect the use of the selected fragment 

If, at step 504, the scenario generator 206 cannot select a grammar input because all the 
available inputs have already been considered in that state, the scenario generator 206 
examines the second list at step 518 to determine whether any dialog states exist in which 

10 not all available inputs have been considered. If no such states exist, then the third list is 
saved as example interactions and the scenario generator 206 hangs up. Otherwise, if such 
states do exist, an unused state is selected at step 516 using the procedure described below, 
and the scenario generator generates a preamble at step 514. The preamble is a sequence of 
events that moves the dialog from the state in which all inputs have been considered into 

15 the selected, state in which not all inputs have been considered. Once the preamble has 
been executed or played back (at step 512) and the dialog is in the new state, the scenario 
generator 206 can resume its random walk. 

Rather than calculating a preamble from a Markov model, the scenario generator 206 uses 
20 a "lazy learning" technique to determine a preamble. A list of target states (Tl) is 

determined from the scenario generator's two lists of valid speech recognition events. 

These states are states representing prompts in which not all inputs have been considered. 

Then, a list of desirable next inputs (NI) is generated from the two lists of valid speech 

recognition events. These inputs are valid inputs in that state in which the response of the 
25 dialog system is expected to be the same regardless of the previous prompt played. 

The scenario starts with the assumption that the dialog system responds to all global 
grammars the same way regardless of the dialog state. For instance, after the speaker says 
"quit" the dialog system may quit. The scenario generator can determine from an 
30 unsuccessful attempt at a preamble that this is not always true. For instance, the dialog 
system's response to phrases such as "repeat" and "help" are highly local, despite the 
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grammar being global. Also, not all states have global fragments active: some states are 
tagged as "Modal", and in such states global fragments are disabled. In addition, the 
response to some global events may depend on quite complicated preconditions. For 
instance, in a purchasing application, the dialog system may respond to a "quit" input by 
5 asking the speaker whether they want to pay for the goods they have ordered, or quit 
regardless. This response would only occur if the speaker has ordered without confirming 
the purchase something prior to saying "quit". 

A sequence of previously executed events is then extracted from the list of example 
10 interactions, that either : 

(i) moves the dialog from the current state to a state in the list Tl ; 

(ii) defines a sequence from an input contained in NI to a state in the list Tl ; 

(iii) defines a sequence from the current state to the end state, plus a sequence from 
the beginning of a call to a state in the list Tl ; or 

15 (iv) defines a sequence from an input contained in NI to the end state, plus a 
sequence from the beginning of a call to a state in the list Tl . 
This sequence of previously seen events is used as the preamble. 

Because the list of valid speech recognition events for each prompt is populated by 
20 experimentation on the part of the scenario generator 206, it only includes a reference to a 
state if it knows how to reach that state from the beginning of a call. In addition, by the 
time the scenario generator 206 reaches a state in which all inputs have been considered, it 
is most likely to have identified a phrase that enables the speaker to end the call. As a 
result, the list of example interactions is highly likely to contain enough information to 
25 determine at least a preamble of the fourth type listed above. The preamble is determined 
by searching through the list of example interactions from top to bottom, one event at a 
time, searching for possible preambles. A count is stored that lists the shortest preamble 
calculated so far. If a preamble is partially identified that is already longer than the 
previously found preamble, that preamble is rejected. As a result, the time required for 
30 determining the shortest preamble is approximately proportional to the length of the data 
contained in the list of example interactions. 
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While a preamble is being played back at step 512, the scenario generator 206 checks that 
the sequence of events being observed is the same as that predicted for the preamble. 
However, only the event type is examined; the slots and actual text are ignored. When the 
preamble defines a spoken input, the scenario generator 206 replays them from the 
5 preamble. The scenario generator 206 records the actual sequence of events and adds them 
to the list of example interactions, regardless of whether it is in random walk or playback 
mode. 

If a preamble fails to behave as predicted, it will usually be because the dialog does not 
10 respond to a speech event in the same way regardless of the previous prompt played. As a 
result, the list of valid speech recognition events for each prompt can be updated, and a 
new preamble identified. If a preamble cannot be identified, the scenario generator 206 
saves the list of example interactions and the scenario generation process terminates. 

15 When generating mixed initiative example interactions, the scenario generator 206 
randomly generates responses by selecting the least used response to any question and by 
selecting grammar rules that are likely to produce phrases that have the largest number of 
slots. The scenario generates as many events in the mixed initiative input as there are 
events in the menu driven inputs. 

20 

One limitation of the process described above is that some states can be "hidden" from the 
scenario generator 206. For example, if the dialog requires a four digit PIN to be spoken, 
the scenario generator 206 will randomly generate a PIN, and is unlikely to generate the 
correct PIN. The existence of hidden states can be easily detected by determining the 
25 percentage of the total numbers of prompts and grammars used. The total number of 
prompts and grammar fragments contained within a VXML script can be extracted from 
the script without requiring any understanding of how the VXML calls these fragments. 

One method of overcoming the hidden state problem is to select a larger number of 
30 representative examples of a particular event type. While this method is suitable for 
fragments in which the number of responses is small (such as "Yes" or "No"), it is likely to 
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create an inordinate number of events for fragments in which the number of responses can 
be large (for example, selecting 1000 different PIN numbers.). Instead, the simulator 208 
records and saves the three data structures described above while the simulator 208 is 
interacting with a human. If the percentage of the application covered by the example 

5 interactions generated by the simulator 208 and the scenario generator 206 is too small, the 
developer is asked to take over from the scenario generator 206 and interact with' the 
simulator 208. Once the developer has reached a previously "hidden" state in the dialog, 
the scenario generator 206 takes over from the developer and creates additionahexample 
interactions. In an alternative embodiment, the developer is asked to interact with the 

10 simulator 208. When the developer has completed interacting with the simulator 208, the 
scenario generator 206 is called again. If the scenario generator 206 determines that any 
previously hidden states are now visible, it can then determine a preamble to reach that 
state, and can continue to generate additional example interactions. 

15 Returning to Figure 4, after the example interactions 234 have been created, the developer 
can view and modify them using the scenario editor 210 at step 408. The scenario editor 
210 generates a graphical user interface window 600 displaying the example interactions in 
an editable table, as shown in Figure 6. Each row in the table corresponds to an action 
taken by either the dialog system or the speaker. The first column 602 indicates whether 

20 the action was taken by the dialog system ("MACHINE>") or the speaker interacting with 
it ("HUMAN>"). The second column 604 indicates either the action type or, in the case of 
a verbal action, the words used in the action. The text of a verbal action can be modified by 
the developer. The third column 606 provides any annotations attached to verbal actions. 

25 The developer can edit the contents of the second column 14 to change the words used by a 
human interacting with the dialog system, as listed in the table. In doing so, the developer 
specifies an alternative way in which the statement could be phrased by a human 
interacting with the dialog system. Similarly, the developer can change the words used by 
the dialog system when interacting with a human. Sentences that have been modified by 

30 the developer are annotated in the example interactions 234 as having been modified by 
the developer. 
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After the developer has completed editing the scenario, an "OK" button 608 is selected to 
close the window 600, exit the scenario editor 210, and execute the dialog learner 212. The 
dialog learner 212 extracts all of the sentences in the example interactions 234 that define 
prompts that have been modified by the developer; these are subsequently used as training 

5 sentences for prompt learning. The number of covering prompts is then determined. Every 
covering prompt is then assigned a count of 1. Every prompt that has been modified by the 
developer and has been extracted from the example interactions is then given a count equal 
to the number of covering prompts. Grammar learning is then applied at step 412 to update 
the prompt grammars 230 using grammatical inference, as described in Starkie. The result 

10 is an updated prompt grammar 230 that can generate the training examples in the example 
interactions 234. 

After the prompt grammar 230 has been updated, the prompts in the example interactions 
234 are updated at step 414. The dialog learner 212 processes the prompts in the example 

15 interactions 234, one prompt at a time. If the prompt text has been annotated as having 
been modified by the developer, the prompt text is left unchanged. If the prompt is 
annotated as being generated by the scenario generator 206, then the prompt text can be 
modified as follows. Firstly, the meaning of the prompt as listed in the annotation is used 
along with the prompt grammar 230 to generate a prompt. If the generated prompt is the 

20 same as that listed in the example interaction, no change is made. Alternatively, if the 
generated prompt differs from that in the example interactions 234, then the two prompts 
are parsed using the prompt grammar 230, and the probabilities that the prompt grammar 
230 would generate the two differing prompts are calculated. If the probability of the new 
prompt is greater than or equal to twice the probability of the existing prompt, then the old 

25 prompt is replaced by the new prompt. 

After the example interactions have been modified, the covering prompts 232 are updated 
at step 416. Only covering prompts that do not contain slots, as indicated by the prompt 
annotations, are candidates for updating. As above, using the slots and the prompt 
30 grammar 230, a prompt representing the annotations is generated. If the prompt that is 
calculated using the prompt grammar 230 differs from the old prompt, the probabilities of 
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the prompt grammar 230 generating both the new and old prompt is then calculated. If the 
probability of the new prompt is equal to or greater than the probability of the prompt 
grammar 230 generating the new prompt, then the prompt is updated. The updating of the 
covering prompts 232 is undertaken to reduce the size of the prompt grammar 230 by 
5 removing entries for generating prompts that will never or rarely be generated in practice 
because a higher probability prompt will be generated instead. This removal of rules will 
only occur if the prompt learner 218 is subsequently re-executed. 

If the developer has also modified example phrases of how speakers will interact with the 
dialog system, these example phrases are also added to the list of training phrases for 
1 0 grammar learning. 

As shown in Figure 6, the scenario editor 210 also provides an "Add question and answer" 
button 610 that enables the developer to add one or more questions and answers to the 
dialog at step 410. To add a question and answer, the developer first must select an entry 

15 in the example interaction 234 (as indicated by the shading in Figure 6) at which to insert 
an instance of the question being asked. When the "Add question and answer" button 610 
is selected, a separate popup window 700, as shown in Figure 7, is generated, allowing the 
developer to type in a question and answer. The popup window 700 includes a question 
text area 702 in which the developer can enter the question text. The developer can then 

20 select one of several predefined answers such as context sensitive help, "function not 
available" or "information not available" by selecting a radio button 704 and selecting a 
desired predefined answer from a pull-down answer menu 706. Alternatively, the 
developer can type in the answer to the question in an answer text area 708 and by 
selecting a new answer radio button 710. A question, such as a Frequently Asked Question 
. 25 (FAQ), can be specified as being able to be asked at any point in the dialog by selecting a 
"Question can be asked at any time" checkbox 712. Alternatively, questions can be made 
specific to a particular point in the dialog by unselecting the checkbox 712. The scenario 
editor 210 determines and stores the name of the prompt that directly precedes the point at 
which the question is inserted. If the name of this prompt cannot be determined, the 

30 question is considered to be able to be asked at any time. 
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When converted to VXML, questions are implemented using links. Firstly, each question 
is given a name, based upon the text of the question. To do this, the question text is first 
copied and then all words appearing in a stop list are removed- Stop lists are used in a 
number of information retrieval systems and contain the most commonly occurring words. 

5 Removal of stop words creates a short sentence that is likely to uniquely represent the 
question being asked. Whitespace in the name is then replaced with an underscore 
character. For example, the question "can i take my dog with me" results in a question 
name of "dogJ\ If the question name is not unique, it has an integer appended to it to 
make it unique. A nonterminal of the form ".Question_ n + question name is then created. 

10 Rather than creating any starting rules during the templating process, as described in 
Starkie, an example sentence is added to the grammar learning training set. After grammar 
learning is applied to the training set, the example phrase contained within the grammar 
learning training set can be generalised to include many other phrases. For instance, if the 
single sentence "can i take my dog with me" is included in the training set and the starting 

15 grammar contains the following rules from the grammar extracted from the application 
library 

!GF_IWantTo i was wondering if i could 1 1 
"GFJEWantTo i want to 1 1 
!GF_IWantTo can i 3 1 
20 !GF_IWantTo i was hoping to 1 1 

then the sentence is generalised to other sentences including: "i want to take my dog with 
me" and " i was wondering if i could take my dog with me." 
This generalisation occurs using grammatical inference, as described in Starkie. 
25 Because the prompt played to answer a question is fixed, a single rule is added to the 
prompt grammar 230 by the dialog learner 214. 



The VXML is then extended by the dialog learner 212 as follows. Firstly, a link is added to 
the top of the VXML, e.g. , 

30 

<link e vent = " FAQNAME" > 

<grammar src= f, airline.rulelist#Question_FAQ2STAM£? "/> 

</link> 
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For example, 

<link event="dog_" > 

<grammar src="airline . rulelist#Question_dog_"/> 

</link> 

Code is then added to the VXML 226 to catch the event generated by the link, using the 
following template: 



<catch event=" FAQNAME" > 
10 <prompt> 

<value expr="PromptAnswer_FAQWAM£() M /> 

< /prompt > 
<reprompt/> 
</catch> 

15 

For example, 

<catch event="dog_" > 
<prompt> 

<value expr="PromptAnswer__dog_() n /> 
20 </prompt> 

<reprompt/> 
</catch> 



If the question can be asked at any time, this code is added directly after the declaration of 
25 the link. Alternatively, if the question can only be asked at a specific time in the dialog 
(i.e., after a particular prompt has been played), the dialog learner 212 adds the event catch 
tag directly after the point at which that prompt is played. 

The scenario editor 210 also enables the developer to disallow a person interacting with the 
30 dialog system to jump from one part of the dialog to another. To do this, the developer 
selects a row labelled n HUMAN>" in the scenario editor window 600 and selects a "Delete 
Bad Interaction" button 612. In response, the scenario editor 210 generates a "Deleting 
dialog transition" window 800, as shown in Figure 8. There are many reasons why a 
transition from one part of a dialog to another may be unsatisfactory, most of them difficult 
35 to infer, and always requiring more than one example. In addition to correctly infer the 
reason why a state transition is invalid, all example interactions need to be tagged 
consistently. To overcome these solutions the "Deleting dialog transition" window 800 
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includes four radio buttons 802 to 808, allowing the developer to select one of these 
buttons to indicate a reason why the state transition is considered invalid. Some of these 
reasons can be eliminated by the development system 100 and are therefore not presented 
to the developer. For instance, the option "(X should not have a hot link)" is only presented 
5 to the developer if X has a hot link, and is not presented to the developer if "X" doesn't 
have a hot link. When an "OK" button 810 is selected, the example interactions 234 and 
dialog state machine code 226 are modified by the dialog learner 212. 

Returning to Figure 3, after the dialog application has been simulated and refined to the 
10 developer's satisfaction, the application 236, including the application code 238 and 
application grammars 240, is generated from the finalised state machine 226 and grammars 
228 and 230 by the code generator 216 at step 308. The application 236 is then installed 
on the IVR 1Q2 to provide an executable dialog system. 

15 Many modifications will be apparent to those skilled in the art without departing from the 
scope of the present invention as herein described with reference to the accompanying 



drawings. 



20 



DATED this 6* day of September 2002 
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By its Patent Attorneys 
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APPENDIX A. Example Interaction Notation 

The following notation is described using Backus Naur Form 
Examplelnteraction : MenuDrivenCails MixedlnitiativeCalls ; 

5 

MenuDrivenCails : 

MenuDrivenMarker ListOfCalls | 
MenuDrivenMarker ; 

10 MenuDrivenMarker : c @startmenudriven:: V; 

MixedlnitiativeCalls : 

EndMenuDrivenMarker ListOfCalls | 
EndMenuDrivenMarker 

15 

EndMenuDrivenMarker : 6 @endmenudriven: : 1' ; 

ListOfCalls : 

ExampleCall j 
20 ExampleCall ListOfCalls 

ExampleCall : 

EventList; 

25 EventList : 

CallEvent | 
CallEvent EventList; 



CallEvent : 

30 Eventld WordsSpoken Integer Tags 
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Eventld : 

♦@' GrammarName '::' TopLevelName | 
'@' EventName'::' ; 

5 WordsSpoken : 
I 

Words | 

Words WordsSpoken; 

10 Tags : 

I 

KeyValuePair | 
KeyValuePair Tags; 

15 KeyValuePair : 

Key Value; 

GrammarName : (any string of characters); 
TopLevelName : (any string of characters); 
20 EventName : (any string of characters); 
Integer : (an integer); 
Words : (any string of characters); 
Key : (any string of characters); 
Value : (any string of characters); 
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Appendix B. Example Interaction 

©startmenudriven: : 1 
©startcall: : 1 1 

©airline. prompts: : .Speaker__WELCOME welcome to the airline application. 1 
5 ©airline. prompts: : .Menu_main_menu please say one of timetables or pricing 
1 

©airline: : . Menu_main_menu_timetables timetables 1 

©airline, prompts :: . Ask__timetablesf orin_destination_city please say the 
destination city 1 
10 ©airline: : . Quest ion__dog_ id like to take my dog with me 1 

©airline. prompts :: .Answer_dog_ yes you can take your dog with you. 1 
©airline. prompts : : . As k_time tables form_destinat ion_city please say the 
destination city 1 

©airline: : . Question_operator someone please 1 
15 ©airline, prompts :: . Answer_operator i'm sorry but that functionality is 
not available in this service 1 

©airline. prompts :: .As k_timetablesform_destination_city please say the 
destination city 1 
©airline: : .Help what can i say 1 
20 ©airline. prompt s : : . Help_Ask_timetablesf orm_destination_city please say 
the destination city, for example, aberdeen 1 
©airline: : . As k_time tables form_destination_city london 1 
destination^city^london" 

©air line. prompts: : . As k_time tables form_city_of ^departure please say the 
25 city of departure 1 

©airline ::. Form_pricingf orm pricing 1 

©airline .prompts :: . As k_pr icing form_destination_city please say the 
destination city 1 

©airline: : . Askj>ricingform__destination_city narrabri 1 
30 destination__city="narrabri" 

©airline. prompts: : . Ask_pricingf orm_city_of_departure please say the city 
of departure 1 

©airline :: .Questionjiello hi there urn 1 
©airline .prompts :: . Answerjiello yes hello 1 
35 ©airline. prompts: : . Ask j?r icing form_city_of ^departure please say the city 
of departure 1 
©airline: : .Quit bye 1 

©airline. prompts: : .Goodbye thank you for using this application, goodbye. 
1 

40 ©endcall:: 1 1 

©startcall: : 1 1 

©airline. prompts: : . Speaker_WELCOME welcome to the airline application. 1 
©airline. prompts: : . Menujnain_menu please say one of timetables or pricing 
1 

45 ©airline: : .Menu_main_menu_pr icing pricing 1 

©airline. prompts :: . Ask_pricingform__destination_city please say the 
destination city 1 

©airline: :. Format irae tables form timetables 1 

©airline. prompts :: .As k_timetablesform_destination_city please say the 
50 destination city 1 

©airline: : .Repeat what did you say 1 

©airline. prompts: : . Ask_timetablesf orm_destination_city please say the 
destination city 1 

©airline: : .As k_timetablesf orm_destination_city london 1 
55 destination_city= rf london" 
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©airline. prompts: : .As k_timetablesform_city_of_departure please say the 
city of departure 1 

©airline: : . Ask_timetablesform_city_of_departure coolangatta 1 
city_of_departure»"coolangatta" 
5 ©airline. prompts: : . As k_time tables form_date please say the date 1 
©airline: : . As k_time table sform_date last Wednesday 1 
date.day_of_week="wednesday" date .modif ier="last" 

©airline, prompts: : . As k_time tables format ime please say the time 1 
©airline: : .As k_timetablesform__time half past twelve in the evening 1 

10 time.hours«12 time . am_or_pm=pm time .minutes=30 

©airline. prompts: : .Conf irm_sayresultstimetables the flight number is 
undefinedundefinedtwo oh oh the destination city is london the city of 
departure is coolangatta the date is last Wednesday the time is twelve 
thirty in the afternoon 1 

15 ©airline. prompts: : .Menujmainjraenu please say one of timetables or pricing 
1 

©airline:-: . Quest ion__ hello hi there urn 1 
©airline. prompts: : . Answer__hello yes hello 1 

©airline. prompts: : . Menu_main_menu please say one of timetables or pricing 



©airline: : . Question_hello yeah well then 1 
©air line. prompts : : . Answer_hello yes hello 1 

©airline. prompts: : . Menu_main_menu please say one of timetables or pricing 



25 ©airline: : . Form_pr icing form pricing 1 

©airline .prompts :: . Ask_pricingf orm_destination_city please say the 
destination city 1 

©airline: : . Ask_pricingf orm_destination_city narrabri 1 
destination_city= "narrabri" 
30 ©air line. prompts :: . Ask jpr icing form_city_of ^departure please say the city 
of departure 1 

©airline: : .As kj?ricingform_city_of_departure nine hundred 1 
city_of_departure«"900" 

©air line. prompts: : . Askjpricingf orm_ticket_class please say the ticket 
35 class 1 

©airline: : .Questionjtiello okay hi uh 1 
©airline. prompts: : . Answerjiello yes hello 1 

©airline. prompts :: .Askjpricingf orm_ticket_class please say the ticket 
class 1 

40 ©airline: : .Askjpricingf orm_ticket_class economy 1 ticket_class="economy" 
©airline. prompts: : . Conf irm_sayresultspricing the destination city is 
narrabri the city of departure is nine oh oh the ticket class is economy 
the price is five hundred dollars 1 price . dollars=500 
destination_city="narrabri" tic ket_class=" economy" price . cents=0 

45 city_of_departure="900 tf 

©airline. prompts: : . Menu_main_menu please say one of timetables or pricing 
1 

©airline: :. Form_pricingf orm pricing 1 

©airline. prompts: : . Ask_pricingf orm_destination_city please say the 
50 destination city 1 

©airline: : . Ask_pricingf orm_destination_city narrabri 1 
destination_city= "narrabri" 

©airline. prompts: : .Askjpricingf orm_city_of_departure please say the city 
of departure 1 
55 ©airline: : . Quest ion_hello hi 1 

©airline .prompts :: . Answer__hello yes hello 1 



20 
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i 




-27- 



Qairline -prompts :: .Askjpricingform_city_of_departure please say the city 
of departure 1 
©hangup: : 1 1 
©endmenudriven: : 1 
5 ©startcall: : 1 

©airline, prompts: : . SpeakerJtfELCOME welcome to the airline application.- 1 
©airline. prompts: : . Menu_main_menu please say one of timetables or pricing 
1 

©airline: : . Form_time tables form timetables 1 
10 ©airline. prompts: : .Ask_timetablesform_destination_city please say the 
destination city 1 

©airline: : . Ask_timetablesf orm_destination_city Portland oregon 1 
destination_city="portland oregon" 

©airline. prompts: : . Ask_timetablesf orm_city_of_departure please say the 
15 city of departure 1 

©airline: : . Ask_timetablesf orm_city_of_departure suva 1 
city_of_departure»"suva" 

©airline. prompts: : .As k_t ime table sform_date please say the date 1 
©airline: : . Ask_timetablesf orm_date the ninth of july 1 date.day«9 
20 date. months" july" 

©airline. prompts: : . As k_time table sform_time please say the time 1 
©airline: : . Ask_timetablesf brm_time thirteen twenty eight 1 time.hours=13 
time . am_or_pm=pm time .minutes -2 8 

©airline. prompts: : .Conf irm_sayresultstimetables the flight number is 
25 undefinedundefinedtwo oh oh the destination city is portland oregon the 
city of departure is suva the date is ninth of july the time is thirteen 
twenty eight 1 

©airline. prompts: : .Menujnain_menu please say one of timetables or pricing: 
1 

30 ©airline: : . Form__t ime tables form timetables 1 

©airline. prompts: : . Ask_timetablesf orm_destination_city please say the 
destination city 1 

©airline: : . Form_pricingf orm pricing 1 

©airline. prompts: : . Ask_pricingf orm_destination_city please say the 
35 destination city 1 

©airline: :. Form_timetablesf orm timetables 1 

©airline. prompts: : . As k_time table sform_destina tion_city please say the 
destination city 1 

©airline: : . Ask_timetablesf orm_destination_city broome 1 
40 destination_city= r, broome" 

©airline. prompts: : . As k_time tables form_city_of ^departure please say the 
city of departure 1 

©airline: : . As k_time table sform_city_of_depHrture albury 1 
city_of_ departure^'albury" 
45 ©airline. prompts :: .Ask_timetablesform_date please say the date 1 
©airline: : . As k_time table sform_date twenty fifth of the month 1 
date.day=25 

©airline. prompts: : . Ask_timetablesf orm_time please say the time 1 
©airline: : . As k_time tables form_time twelve fifty at night 1 time.hours=12 

50 time . am__or_pm-pm time .minutes=50 

©airline. prompts: : .Conf irm_sayresultstimetables the flight number is 
undefinedundefinedtwo oh oh the destination city is broome the city of 
departure is albury the date is the twenty fifth day of the month the 
time is twelve fifty in the afternoon 1 

55 ©airline .prompts: : .Menu__main_menu please say one of timetables or pricing 
1 

©airline: :. Form_pricingf orm pricing 1 
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©airline. prompts: : .Ask_pricingf orm_destination_city please say the 
destination city 1 

©airline: : .As kjpricingform_destination_city dublin 1 
destination_city="dublin" 
5 ©air line. prompts: : . As k_pr icing form_city_o ^departure please say the city 
of departure 1 

©airline: : . As k_pr icing form_city_of _departure h seventeen three thousand 1 
city_of_departure=*"hl73000" . 
©airline, prompts: : .As kjpricingform_ticket_class please say the ticket 
10 class 1 

©airline: : .As k_pricingf orm_ticket_class economy 1 ticket_class=" economy" 
@airline. prompts :: . Con firm_sayre suit spf icing the destination city is 
dublin the city of departure is h one seven three oh oh oh the ticket 
class is economy the price is five hundred dollars 1 price . dollars=500 
15 destination_city="dublin" ticket_class=" economy" price. cents=0 
city_of_departure«"hl73000" 

©airline. prompts :: .Menu_main_menu please say one of timetables or pricing 
1 

©airline: : . Form_time tables form timetables 1 
20 ©airline. prompts: : . Ask^timetablesf orm_destination_city please say the 
destination city 1 

©airline: : . Ask_timetablesform_destination_city torn price 1 
destination_city="tom price" 

©airline. prompts: : . As k_time table sform_city__o f__departur e please say the 
25 city of departure 1 

©airline: : . Ask_timetablesform_city_of_departure belfast 1 
city_of_departure="belfast" 

©airline. prompts: : . As k_time tables form_date please say the date 1 
©airline: : . As k_t ime tables form_date this coming Wednesday 1 
30 date.day_of_week="wednesday" 

©airline. prompts :: . As k_time tables form_time please say the time 1 
©airline: : .As k_timetables format ime one oh one ami time . hour s=l 
time . am_or_j?m=am time .minutes=l 

©airline .prompts :: . Conf irm_sayresultstimetables the flight number is 
35 undefinedundefinedtwo oh oh the destination city is torn price the city of 
departure is belfast the date is Wednesday the time is one oh one in the 
morning 1 

©air line. prompts: : . Menu_main_menu please say one of timetables or pricing 
1 

40 ©airline: :. Formjpricingf orm pricing 1 

©airline. prompts :: .As k_j?ricingform_destination_city please say the 
destination city 1 

©airline: : . Ask_pricingf orm_destination_city detroit 1 
destination_city="detroit" 
45 ©airline. prompts :: .As k_pricingf orm_city_of_departure please say the city 
of departure 1 

©airline :: .As k_pricingf orm_city__of_departure five hundred 1 
city_of ^depart ure= "500" 

©airline. prompts :: . Ask_pricingform_ticket_class please say the ticket 
50 class 1 

©airline: : . Ask_pricingf orm_ticket_class business 1 
ticket_class="business " 

©airline. prompts : : .Conf irm_sayresultspricing the destination city is 
detroit the city of departure is five oh oh the ticket class is business 
55 the price is five hundred dollars 1 price. dollars»500 

destination_city= M detroit" ticket_class="business" price . cents=0 
city_of_departure="500" 
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©airline. prompts: : . Menu_main_menu please say one of timetables or pricing 
1 

©airline: : . Form_time tables form timetables 1 

©airline. prompts: : .As k_timetablesf orm_destination_city please say the 
5 destination city 1 

©airline: : . As k_time tables form_destinat ion_city taree 1 
destination_city="taree" 

©airline, prompts: : . As k_time table sform_city_of_departure please say the 
city of departure 1 
10 ©airline:!: . Ask_timetablesf orm_city_of_departure paris 1 
city_of_departure="paris" 

©airline, prompts: : . As k_time table sform__date please say the date 1 
©airline: : . Ask_timetablesf orm_date friday 1 date . day_of_week==" friday" 
©airline. prompts: : . As k_time tables format ime please say the time 1 
15 ©airline: : . As k_time table sform_time twenty two twenty four 1 time .hour s«2 2 
time . am_or__pm=pm time. minute 3=2 4 

©airline, prompts: : . Con firm_sayr e suits timetables the flight number is 
undefinedundefinedtwo oh oh the destination city is taree the city of 
departure is paris the date is friday the time is twenty two twenty four 
20 1 

©airline. prompts: : .Menujnainjnenu please say one of timetables or pricxng 



1 

©hangup: : 



1 
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